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A METHOD FOR COMPUTING OPTICAL FLOW UNDER THE EPI POLAR 

CONSTRAINT 

TECHNICAL FIELD OF THE INVENTION 

The present invention is directed, in general, to 
processing an image sequence for modeling of the content or 
modification of a perspective for the scene within the 
image sequence and, more specifically, to computing optical 
flow by imposing a global, non-heuristic, geometrical 
constraint, either alone or in combination with heuristic 
constraints . 

BACKGROUND OF THE INVENTION 

For consecutive image sequences such as found in video 
presentations, optical flow is the velocity field which 
warps one image into another (usually very similar) image. 
Optical flow is computed using the well-known optical flow 
equation : 

I x u + I y v + I t = 0 (1) 
where I x , I , and I t represent the spatio-temporal deriv- 
atives of image intensity (all known) at a given point A 
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and (u,v) represents the (unknown) image velocity across an 
image sequence at the same point. The system defined by 
equation (1) is obviously unconstrained since at least one 
more constraint is required to uniquely solve for image 
5 velocity (u,v) . It should be noted that equation (1) defines 

a line in (w,v) space, as illustrated in FIGURE 5A. 

A variety of methods have been employed or proposed 
S for determining the second constraint required for unique 

jC solution of the system of equation (1) . However, a common 

1(£ feature of the many different methods for introducing the 

^ second constraint is that the second constraint is local 

y, and heuristic. For example, one technique assumes the 

L;1 image velocity (u,v) is locally uniform and determines the 

N ; velocity by minimizing the total error in point A by, for 

15 example, finding the image velocity (w,v) which minimizes the 

expression : 

^(I x u + I y v + I t f (2) 

Q 

where Q denotes the region over which the image velocity is 
assumed constant. A geometric interpretation of this 
20 (least squares) minimization problem is illustrated in 

FIGURE 5B. Basically, the vector (u,v) is found as a point 

with the minimum total distance from lines I v I 2 J^...J n . 
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The main disadvantages of this method are: (1) since 
the image velocity is computed for each point, the method 
is computationally expensive; and (2) if the image velocity 
or variation between sequential images is low (i.e., 
5 I x *I y *0) , the image velocity either can not be determined 

or is very sensitive to noise. Additionally, special 
problems arise in the presence of strong noise and/or 
yQ multiple motions present in the vicinity of image point 

yj ( x >y) r as illustrated in FIGURE 5C. 

To overcome these disadvantages, a class of methods 
^" referred to as "sparse optical flow" computation has been 

y introduced. Such methods are typically performed utilizing 

yp; two steps: (1) find points in the image with high inform- 

H ! ation content (typically, high variation in I x , I); and 

5 (2) compute optical flow at those points utilizing point 

matching between consecutive image frames (the phrases 
u sparse optical flow" and "point matches" will be used 
interchangeably herein) . While these methods are faster 
and more reliable than "dense" optical flow methods, the 
D disadvantage is that the optical flow is computed only at a 

small number of points. 
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There is, therefore, a need in the art for a method of 
combining the benefits of both dense and sparse optical 
flow without incurring the respective disadvantages. 
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To address the above-discussed deficiencies of the 
prior art, it is a primary object of the present invention 
5 to provide, for use in video system, a method of computing 

optical flow for an image sequence by imposing a global, 
non-heuristic, geometrical constraint. Point matches 

between images within the image sequence are identified by 
sparse optical flow computation and employed to compute a 

ltpj fundamental matrix for the epipolar geometry, which in turn 

is employed to derive an epipolar geometry constraint for 
computing dense optical flow for the image sequence. The 
Ui epipolar geometry constraint may further be combined with 

local, heuristic constraints or robust statistical methods. 

15 Improvements in both accuracy and performance in computing 

optical flow are achieved utilizing the epipolar geometry 
constraint . 

The foregoing has outlined rather broadly the features 
and technical advantages of the present invention so that 
20 those skilled in the art may better understand the detailed 

description of the invention that follows. Additional 
features and advantages of the invention will be described 
hereinafter that form the subject of the claims of the 
invention. Those skilled in the art will appreciate that 
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they may readily use the conception and the specific 
embodiment disclosed as a basis for modifying or designing 
other structures for carrying out the same purposes of the 
present invention. Those skilled in the art will also 
realize that such equivalent constructions do not depart 
from the spirit and scope of the invention in its broadest 
form. 

3 Before undertaking the DETAILED DESCRIPTION OF THE 

J:;! INVENTION below, it may be advantageous to set forth 

2: definitions of certain words or phrases used throughout 

^ this patent document: the terms "include" and "comprise, " 

«§, as well as derivatives thereof, mean inclusion without 

5 limitation; the term "or" is inclusive, meaning and/or; the 

3 phrases "associated with" and "associated therewith," as 

well as derivatives thereof, may mean to include, be 
included within, interconnect with, contain, be contained 
within, connect to or with, couple to or with, be 
communicable with, cooperate with, interleave, juxtapose, 
be proximate to, be bound to or with, have, have a property 
of, or the like; and the term "controller" means any 
device, system or part thereof that controls at least one 
operation, whether such a device is implemented in 
hardware, firmware, software or some combination of at 
least two of the same. It should be noted that the 
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functionality associated with any particular controller may 
be centralized or distributed, whether locally or remotely. 
Definitions for certain words and phrases are provided 
throughout this patent document, and those of ordinary 
skill in the art will understand that such definitions 
apply in many, if not most, instances to prior as well as 
future uses of such defined words and phrases. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention, and the advantages thereof, reference is now 
5 made to the following descriptions taken in conjunction 

with the accompanying drawings, wherein like numbers 
designate like objects, and in which: 

FIGURE 1 depicts a block diagram of a video system 
employing a method of computing optical flow under an 
epipolar geometry constraint according to one embodiment of 

ff : 

y the present invention; 

\*> FIGURE 2 depicts image data employed in computing 

p optical flow under an epipolar geometry constraint 

O according to one embodiment of the present invention; 

15 FIGURE 3 illustrates a geometric interpretation of 

computing optical flow in the presence of strong noise 
and/or multiple motions in the vicinity of the subject 
image point utilizing robust statistical methods under an 
epipolar geometry constraint according to one embodiment of 
2 0 the present invention; 

FIGURE 4 is a high level flowchart for a process of 
computing optical flow under an epipolar geometry 
constraint according to one embodiment of the present 
invention; 
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FIGURES 5A through 5C are geometric interpretations of 
various optical flow problems; and 

FIGURES 5D and 5E depict models for computing epipolar 
geometry. 
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FIGURES 1 through 4, discussed below, and the various 
embodiments used to describe the principles of the present 
invention in this patent document are by way of 
illustration only and should not be construed in any way to 
limit the scope of the invention. Those skilled in the art 
will understand that the principles of the present 
invention may be implemented in any suitably arranged 
device . 

FIGURE 1 depicts a block diagram of a video system 
employing a method of computing optical flow under an 
epipolar geometry constraint according to one embodiment of 
the present invention. Video system 100 includes a video 
receiver, which may be a computer or other data processing 
system; a satellite, terrestrial or cable broadcast 
television receiver; a set top box; or a video game unit. 
Video receiver 101 has an input 103 for receiving a 
sequence of images and an optional output 104 for 
transmitting processed image data to, for example, a 
display or recording device. 

Video receiver 101 in the exemplary embodiment 
includes an image processor 104 computing optical flow for 
the received sequence of images as described in further 
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detail below. Video receiver 101 may also optionally 
include a memory 105 for storing the received sequence of 
images, computed optical flow information, or other data, 
and a display 106 for displaying images or image sequences 
modified utilizing the computed optical flow information. 
For example, image processor may be employed in depth 
reconstruction, body modeling, or image morphing for three 
dimensional (3D) television, and may generate altered image 
sequences based on received image sequences to display a 
scene from a different perspective. 

Those skilled in the art will recognize that the 
complete structure and operation of a video receiver is not 
depicted or described herein. Instead, only so much of the 
structure and operation of a video receiver as is either 
unique to the present invention or necessary for an 
understanding of the present invention is depicted in the 
figures and described. The remainder of the structure and 
operation of a suitable video receiver may be conventional, 
although in the present invention the video receiver 
computes optical flow information for the received image 
sequence as described in further detail below. 

FIGURE 2 depicts image data employed in computing 
optical flow under an epipolar geometry constraint 
according to one embodiment of the present invention. In 
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computing optical flow according to the present invention, 
video receiver 101 receives a sequence of images 200, and 
either receives or computes (using, e.g., image processor 
104) a tabulation 2 01 of point matches, corresponding image 
pixels, between images within the sequence. 

In the present invention, optical flow for the 
received sequence of images 200 is computed by first 
performing the sparse optical flow computation described 
above. That is, the points in the image (s) with the 
richest information content (e.g., high variation in I x , I 
such as corners) are identified and matched between images 
within the sequence. The identified point matches are then 
employed to compute the epipolar geometry relating the 
views between consecutive frames, described by a funda- 
mental matrix F . Normally at least seven point matches are 
required to compute the fundamental matrix F relating all 
pixels within two images for the image sequence. 

In computing the fundamental matrix, a pinhole camera 
model illustrated in FIGURE 5D is employed for the first 
view/frame to define the location within the camera 
reference frame (x , y , and z, where z is the optical axis) 
of an image point m representing the intersection of a line 
containing a three dimensional world point W and the 
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camera's optical center C with an image plane R . The 
projection of the optical center C for a similar pinhole 
camera representing the second view/frame onto the image 
plane R of the first camera or view/frame, illustrated in 
FIGURE 5E, is epipole e, and epipole e' is similarly the 
projection of the optical center C onto the image plane R' 
of the second camera or view/frame. The image points m 
(defined by coordinates x,y ) and rri (defined by coordinates 
x\y r ) , which represent the projection of the world point W 
onto the image planes R and R f , respectively, constitute a 
conjugate pair, and a fundamental matrix F relating such 
conjugate pairs may be computed from at least seven image 
point matches, in accordance with the known art (see, e.g., 
"Epipolar Geometry and the Fundamental Matrix" in R. 
Hartley & A. Zisserman, Multiple View Geometry in Computer 
Vision pp. 219-242 (Cambridge University Press 2000), 
incorporated herein by reference) . Additionally, epipoles 
e and e f and image point m form a plane, for which the 
intersection with image plane R is epipolar line L ; 
similarly, epipoles e and e' and image point m r form a 
plane, for which the intersection with image plane R r is 
epipolar line V . 
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Referring back to FIGURE 2, the fundamental matrix F 
imposes a global constraint on point matches (x,y) and (x',/) 
given by: 



The geometric interpretation of this constraint is that the 
point matches (x\y r ) for the point at coordinates (x,y) must 
lie on the epipolar line V . By substituting x r = x + u and 
y' = y + v, equation (3) may be written as: 



where a xy , b xy and c Xty (or, alternatively, simply a, b and 
c) are obtained from the fundamental matrix F and point 
coordinates (x 9 y) . Optical flow may therefore be calculated 
from the combination of equations (1) and (4) . Since both 
equations are linear, optical flow is completely 
determined. Dense optical flow (i.e., optical flow at each 
pixel location within at least one of the two images) is 
therefore easily computed utilizing the epipolar geometry 
constraint of equation (4) in combination with the basic 
optical flow equation (1) . 

In addition, the epipolar geometry constraint defined 
by equation (4) may be utilized to improve computation of 



[x y l]F y' =0 . 



(3) 



0 



(4) 
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optical flow by any known methods, including, for example, 
the minimization method described above. Combining 
equations (2) and (4), optical flow may be computed by 
minimizing, in u t the expression: 

Z((bl x -al y )u + cl y -bl t ) 2 

x,yeO 

and then finding v as: 

c-au 
v = . 

b 

Another example of the usefulness of the epipolar 
geometry constraint involves computation of optical flow 
utilizing robust statistics . Referring back to the 

geometric interpretations of equations (1) and (2) 
illustrated in FIGURES 5A through 5C, the problems arising 
from the presence of strong noise and/or multiple motions 
in the vicinity of point (x,y) prevent effective use of the 
least squares minimization method to obtain the motion 
vector (m,v) . Robust statistical methods, such as Least 
Median of Squares (LMedS) , RANdom SAmple Consensus 
(RANSAC) , etc., are required. However, such robust 

statistical methods are typically exponential in the size 
of the search space and, in general, are not easy to 
implement for a search space size greater than one. 
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Computation of optical flow involves a search space size of 
two, namely u and v . 

FIGURE 3 illustrates a geometric interpretation of 
computing optical flow in the presence of strong noise 
and/or multiple motions in the vicinity of the subject 
image point utilizing robust statistical methods under an 
epipolar geometry constraint according to one embodiment of 
the present invention. By imposing the epipolar geometry 
constraint of equation (4) , optical flow vectors (u,v) are 
determined by the intersection between the optical flow 
constraint lines I v I 2 ,I 3 ,... 9 I n and the epipolar line defined by 
equation (4) . Optical flow is thus easily determined 
utilizing robust statistical methods such as, for instance, 
Least Median of Squares by: (1) finding, for each line I lt 
the intersection (m.,v.) of the respective line with the 
epipolar line e; finding the horizontal (or vertical, or 
any other parameterization) component of the optical flow 
vector (as u = median(u i ) for Least Median of Squares); and 

t=l,.. ,n xi 

then finding the vertical component of the optical flow 
vector as 

c-au 
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The above discussions assume that no motion relative 
to an image frame of reference is occurring by objects 
within the image, and that motion is due only to camera or 
view movement. However, if the images contain moving 
objects (the validity of the computed fundamental matrix 
may be assumed to be accurate despite such motion if the 
image is dominated by the stationary background to the 
moving objects since robust statistical methods known in 
the art may be utilized in such cases) , similarity of 
pixels which should match based on the derived optical flow 
may be evaluated, and each pixel assigned a confidence 
level. Pixels associated with moving objects in such a 
situation will receive low confidence levels, while pixels 
relating to the background will receive high confidence 
levels. Pixels having a high confidence level may then be 
disregarded, and the remaining pixels (assumed to relate to 
moving objects) employed to compute a separate epipolar 
geometry for computing dense optical flow for those pixels. 

FIGURE 4 is a high level flowchart for a process of 
computing optical flow under an epipolar geometry 
constraint according to one embodiment of the present 
invention. The process 400 begins with receipt of an image 
sequence (step 401) for which optical flow information is 
needed. Point matches between images within the image 
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sequence are first computed utilizing sparse optical flow 
computation (step 4 02) , and are employed to determine the 
fundamental matrix of correlation between images within the 
image sequence (step 403) . 
5 The fundamental matrix, in turn, is employed to derive 

an epipolar geometry constraint for optical flow 
computation, and optical flow is computed for each pixel 
^| within at least one of the image under the epipolar 

J geometry constraint (step 404) . Computation of optical 

1( Ei flow under the epipolar geometry constraint may also 

y x involve use of local, heuristic constraints such as least 

y> squares minimization or robust statistical methods. The 

p process then becomes idle (step 405) until another image 

w sequence is received for computation of optical flow 

15 information . 

The present invention computes optical flow by 
imposing a global, non-heuristic, geometrical constraint 
derived from epipolar geometry for the underlying image 
sequence on the basic optical flow equation. The epipolar 
2 0 geometry constraint may be combined with heuristic 

constraints. In any event, improved accuracy and 

performance in computation of the optical flow vector is 
achieved. 
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It is important to note that while the present 
invention has been described in the context of a fully 
functional system, those skilled in the art will appreciate 
that at least portions of the mechanism of the present 
invention are capable of being distributed in the form of a 
machine usable medium containing instructions in a variety 
of forms, and that the present invention applies equally 
regardless of the particular type of signal bearing medium 
utilized to actually carry out the distribution. Examples 
of machine usable mediums include: nonvolatile, hard-coded 
type mediums such as read only memories (ROMs) or erasable, 
electrically programmable read only memories (EEPROMs) , 
recordable type mediums such as floppy disks, hard disk 
drives and compact disc read only memories (CD-ROMs) or 
digital versatile discs (DVDs) , and transmission type 
mediums such as digital and analog communication links. 

Although the present invention has been described in 
detail, those skilled in the art will understand that 
various changes, substitutions, variations, enhancements, 
nuances , gradations , lesser forms , alterations , revisions , 
improvements and knock-offs of the invention disclosed 
herein may be made without departing from the spirit and 
scope of the invention in its broadest form. 



